Papers with sample weighting
Efficient Annotator Reliability Assessment with EffiARA (2025.acl-demo)
Copied to clipboard
| Challenge: | Obtaining annotations from experts is ideal, but this expertise is logistically and financially costly. |
| Approach: | They propose an annotation framework that supports the whole annotation pipeline from understanding the resources required for an annotation task to compiling the annotated dataset. |
| Outcome: | The proposed framework improves classification performance through annotator-reliability-based soft-label aggregation and sample weighting, and increases agreement among annotators through removal of identifying and replacing an unreliable annotation. |
Efficient Annotator Reliability Assessment and Sample Weighting for Knowledge-Based Misinformation Detection on Social Media (2025.findings-naacl)
Copied to clipboard
Owen Cook, Charlie Grimshaw, Ben Peng Wu, Sophie Dillon, Jack Hicks, Luke Jones, Thomas Smith, Matyas Szert, Xingyi Song
| Challenge: | Misinformation spreads rapidly on social media, confusing the truth and targeting potentially vulnerable people. |
| Approach: | They propose to use inter- and intra-annotator agreement to understand the reliability of each annotator and influence the training of large language models based on annotators reliability. |
| Outcome: | The proposed framework utilises inter- and intra-annotator agreement to understand the reliability of each annotator and influence the training of large language models based on annotators reliability. |
Learning Temporally-Aware Sample Weights for Preference Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for preference optimization rely on static functions of instantaneous model states and ignore temporal learning dynamics. |
| Approach: | They propose a framework that meta-learns adaptive weights using three temporal features: reward margin evolution, learning volatility, and reference deviation. |
| Outcome: | The proposed framework achieves statistically significant improvements over baselines on models ranging from 7B to 70B parameters. |